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Abstract 

This evaluation report summarizes Year 4 of the Math and Science Academy 
(MSA), an initiative of the Northern New Mexico Council on Excellence in Education 
(NNMCEE). The report begins with an overview of the project and its objectives, and 
then outlines the research questions and methods used to carry out the evaluation. 
Findings from the Year 4 evaluation are presented next; the report concludes with 
recommendations and refinements for future years of MSA. 

MSA Project Goals and Objectives 

The Math and Science Academy is an on-going project geared toward the 
development of quality teaching and learning to support student achievement. It is a 
collaborative effort, developed jointly by the Northern New Mexico Council on 
Excellence in Education (NNMCEE) and local school districts (Chama, Espanola, and 
Mora), the Northern Network for Rural Education, the University of California and the 
Department of Energy's Los Alamos National Laboratory (LANL). The program is 
based on a model of school reform that emphasizes professional collaboration and 
research-based approaches to teaching and learning, and uses formative evaluation as a 
guide for on-going program development and refinement. The project was initially 
introduced at the middle school level, in an effort to provide middle school teachers 
with opportunities through which to strengthen instructional practices and increase 
student learning. 

MSA's initial 3-year focus on middle school students and their teachers was 
based on research, longitudinal studies, and statistical analyses of achievement scores, 
which all indicated the critical need for quality programs and well-prepared teachers to 
support student learning during the challenging years of early adolescence. For Year 4 
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of the project, MSA expanded to elementary teachers and students because of an 
acknowledged need to include the "feeder schools" in the teaching and learning 
processes developed and implemented at MSA middle schools. 

Project Overview 

The goals of the project are ambitious and far reaching — student test scores and 
teacher competency surveys reveal a continuing need for improving student 
achievement and teacher preparation and education in Northern New Mexico. 
Additionally, the Los Alamos National Laboratory, a partner in the project, has an on- 
going interest in providing employment opportunities to the citizens of Northern New 
Mexico, necessitating the cultivation of a well-educated and well-trained workforce. 

Over the past four years, MSA has grown in size and scope: the initial Year 1 
cohort involved 3 schools in 3 districts with 4 teachers at each middle school site, and 
was guided by 2 mentor teachers. Year 4 of the MSA project involved 9 schools in 4 
districts, 53 teachers and 3 mentor teachers. As the project has grown, program goals 
have been refined and more clearly articulated, and MSA members (teachers and 
mentors) have become more savvy and sophisticated in understanding the ways in 
which to maximize project impact. Simultaneously, tools and structures have been 
developed to increase implementation of project goals to support students, teachers, 
administrators, and schools in the project. 



Evaluation and Design Issues 

Years 1, 2, and 3 of the UCLA/ CRESST evaluation of the MSA project were 
designed to describe how the program was implemented, to assess program effects and 
to generate recommendations for the improvement and enhancement of the project. 
Year 4 of the MSA evaluation paralleled the same research questions as Years 1 - 3 of 
the project, and was expanded to include a number of different features designed to 
further strengthen the project and gauge project impact as MSA expanded to 
elementary schools. The following research issues were examined: 



• How did the MSA program evolve? 

• What was the effect of MSA on teaching and learning? 

• How can the program be refined and sustained? 
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• How can the program to be refined to better support and enhance teacher 
professional development, administrative leadership, and student learning and 
achievement? 



Additionally, Year 4 of the project worked to further develop a student database 
to monitor long term, intended effects of the project — longitudinal records of student 
achievement data, student absences, participation in special programs, awards received, 
drop-out rates, and grades. As previous years of the project have shown, these student 
data provide critical information that allow for a better understanding of the long-term 
effects of teachers and school participation in a project such as MSA. 

The evaluation employed a multi-method approach to understand and assess 
program implementation and effects. Surveys, interviews, focus groups, and program 
documents were used as information sources on program implementation and impact. 
We continued to use a formative approach to our research; that is, we conferred 
systematically with project participants throughout the year, and provided important 
information to project members regarding project successes and challenges as issues 
emerged. 

As in previous evaluations. Year 4 research incorporated results and findings 
from Years 1, 2, and 3 of MSA. The first year of the project evaluation was spent in 
gaining understanding of the project and its goals, in developing relationships with 
mentors, project administrators and project members, and helping to the refine the 
overall program theory of action. The Year 2 evaluation helped project leaders to closely 
evaluate the programs' impact on teachers' knowledge, understanding and 
implementation of project goals, and to further implementing the evaluation tools 
developed during prior years of the program. Year 3 further examined the project's 
impact on teachers and students as the program expanded and was more widely 
implemented. The Year 4 evaluation focuses on how MSA has continued to evolve and 
on documenting the ways in which the project expanded at the elementary school level. 
Additional data are presented on administrators and their role in the support and 
development of quality teaching and learning at the MSA sites. 

Method 

Multiple sources of information were used to understand project implementation 
and impact. Table 1 displays the various methods used to gather data during Year 4 of 
MSA. 
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Table 1 

MSA Year 4: Data Sources 



Student Data 


Classroom 

Assignments 


Surveys 


Observations 


Interviews 


•NRTs 


• Science lessons 


• Teachers 


• Classrooms 


• Teachers 


Standardized 


• Math lessons 




• Professional 


• Principals 


achievement 


• Language arts 




development sessions 


• Administrators 


scores (CTB) 


lessons 




• Celebration of 


• Project mentors 


- language arts 

- reading 

- math 

- science 

- social studies 


• Social Studies 
lessons 




Learning 

• Summer Institute 
(2004) 


• Focus groups 



Instruments. Student achievement data were collected by schools on the norm- 
referenced assessment, CTB/Terra Nova, published by McGraw Hill. Longitudinal data 
were analyzed for MSA student performance from the past 2 years (2002-2003 and 
2003-2004) where these data were available. Classroom assignments were collected 
from teachers (and mentors) during the April/ May site visit when classroom 
observations were completed. Assignments represented typical work from all content 
areas. 

Classroom observations gathered data on project goals, including instructional 
approaches, grouping patterns and practices, assessment strategies and teacher-student 
interactions. Post-instructional interviews with teachers included specific questions 
about the lessons observed and general questions about the ways in which MSA 
objectives were evident or implemented in classrooms. 

Surveys were administered at the beginning of the 2004 Summer Institute. The 
teacher survey queried teachers on the ways in which participation in the Math and 
Science Academy influenced teachers' instructional practices, their knowledge and 
understanding of standards-based instruction, use and knowledge of technology, 
willingness to use a variety of instructional settings, assessment practices, and ratings of 
program effectiveness. The survey asked both open-ended questions and questions that 
involved a 5-point rating scale. Most MSA principals were interviewed about the 
program, and answered questions about project successes and barriers and general 
impressions regarding MSA's impact on teachers and students. Copies of all 
instruments developed and used in the evaluation can be found in Appendix A. 
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Project Findings: Year 4 



Findings from Year 4 of MSA are presented in the following sections. 
Demographic information on teacher participants is presented first, followed by general 
demographic information on students and schools. Results in the second section are 
organized around the research questions for Year 4 of MSA. The report concludes with 
additional information on project implementation and impact and recommendations for 
future years of MSA. 



Participants 

MSA Teachers. Table 2 below displays demographic data for the forty-three Year 
4 MSA teachers who completed surveys at the conclusion of the 2003-2004 school year. 
A total of 53 teachers were full participants (defined as participating in the majority of 
MSA requirements) in Year 4 of MSA. Teachers were from 4 districts (including 4 
middle schools and 5 elementary schools) all located in Northern New Mexico. Survey 
data are available for 43 of 53 teachers who were program participants: data are absent 
from 10 teachers who left the project on or before the conclusion of the 2003-2004 school 
year, or who were unable to attend the 2004 Summer Institute. As always, caution 
should be used in interpreting the survey results due to the incomplete data set. 

MSA members who completed the survey included 36 female and 7 male 
teachers of primarily Hispanic, Latino/ a and/ or Spanish American ethnicity (84%); the 
remaining 16% of MSA teachers were white or Native American, or did not specify their 
ethnicity on the survey. In general, MSA teachers were an experienced group with an 
average of 10.9 years of teaching (range: 1-31 years). They held multiple credentials — 
bilingual, single subject, language arts, special education and early childhood 
education — with the majority of teachers holding an elementary credential. Most MSA 
teachers have Bachelor's degrees with teaching credentials and units beyond those 
degrees, and eight teachers have Master's degrees with additional course units of study. 
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Table 2 



Year 4 MSA Teacher Demographic Information 



Variable 


Descriptor 


II 

£ 


Sex 


Male: 


7 




Female: 


36 


Ethnicity 


White: 

Hispanic/Latino/a 


6 




Spanish American: 


35 




Native American: 


1 




Other: 


1 


Highest Degree Received 


Bachelor's Degree: 


9 




Bachelor's + Credential + Units 


24 




Beyond: 


2 




Master's + Units Beyond: 


8 




Doctorate: 


0 




Other: 


0 


Teaching Credential* 


General Elementary: 


27 




General Secondary: 


7 




Special Emergency: 


4 




Multiple Subject: 


3 




Single Subject: 


8 




Bilingual: 


15 




Other: (Lang Arts, TESOL): 


16 


Years of Experience 


Average Number: 


10.9 years 




Range of Years Teaching: 


1-31 


Previous participation in projects 
like MSA 


Number of teachers: yes 


30 


Number of Years in Project 


1 st Year MSA 


32 




2 nd Year MSA 


1 




3rd year MSA 


3 




4 th Year MSA 


7 



*Note: teachers may hold multiple credentials. 



As in previous years of MSA, a number of teachers reported teaching outside 
their content area specialization. This situation represents a continuing and on-going 
challenge for teachers who instruct in content areas for which they have little formal 
training or content knowledge. At some sites, MSA teachers teach three different 
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subjects at three different grade levels. Most teachers (over 60%) reported English as 
the primary language of instruction, with another 22% reporting the use of both English 
and Spanish during instruction. Code switching, at culturally appropriate times, was 
observed in many MSA classrooms. When necessary, MSA teachers often provided 
additional explanations or instructions to students in Spanish with limited knowledge 
of English. Of the 43 teachers surveyed, 70% reported previously participating in reform 
initiatives, including MathStar, Math and Science Reform, and the University of New 
Mexico Academy. 

MSA students. During the fourth year of MSA, approximately 1350 students 
from 9 different schools participated in the project. They ranged in grades from 
Kindergarten through 9 th grade. In total, approximately 680 elementary students 
(Grades K-5), 620 sixth- to eighth-grade students and 60 ninth-grade students were 
participants at schools where teachers were involved with MSA. All 7 th grade students 
participated in MSA at 2 of the project sites. At two schools, all 6 th , 7 th and 8 th grade 
students participated in MSA. At a fourth school (8 th grade only) a subset of students, 
those who participated in Year 3 MSA for their 7 th grade year, continued to be a part of 
MSA for 8 th grade. For the elementary sites, one school had 100% project participation, 
while the percentage and grade level of MSA involvement and participation varied 
considerably at the other elementary schools. 

Student ethnicity was primarily Hispanic/ Latino/ a (70%), with roughly 15% 
Native American and 15% white and/or other ethnicities, although these percentages 
varied from school to school. More than 80% of the MSA student population qualified 
for a free/ reduced lunch program (an indicator of poverty), while more than 30% of the 
total population was identified as English Language Learner (ELL). Table 3 displays 
more specific demographic information about Year 4 MSA students. 
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Table 3 



MSA Year 4: Student Demographics 





School 


School 


School 


School 


School 


School 


School 


School 


School 




A 


B 


C 


D 


E 


F 


G 


H 


I 


School Level 


Middle 


Middle 


Middle 


Middle 


Elem 


Elem 


Elem 


Elem 


Elem 




6 th - 8 th grade 


6 th - 8 th grade 


7 th grade 


8 th & 9 th 


K - 5 


K-5 


K - 6 


K - 6 


K - 6 




Chama 


Mora 


only 
Espan 7 




(Chama) 


(Mora) 


(Fairview) 


(San Juan) 


Pojoaque 


Total Number 
of MSA 
Students 


73 


170 


170 


180 


110 


46 


132 


117 


277 


Grade Level/ 


6 th :26 


6 th :61 


7 th : 170 


8 th : 120 


K 20 


5 lh: 46 


5 th : 60 


5 th : 56 


4 th : 23 


MSA 


7 th :22 


7 th :51 




9 th : 60 


1 st : 21 




6 th : 72 


6 th : 61 


5 th : 56 


Student Count 


8 th :25 


8 th :58 






2 nd : 22 








6% 198 












3 rd : 12 




















4 th : 16 




















5 th : 19 










% Gender 


F: 47% 


F: 48% 


F: 51% 


F: 47% 


F: 48% 


F: 46% 


F: 46% 


F: 52% 


F: 49% 




M: 53% 


M: 52% 


M: 49% 


M: 53% 


M: 52% 


M: 54% 


M: 54% 


M: 48% 


M: 51% 


% Ethnicity 


Latino: 62% 


Latino: 95% 


Latino: 76% 


Latino: 83% 


Latino: 72% 


Latino: 98% 


Latino: 86% 


Latino: 62% 


Latino: 70% 




Native 


Native 


Native 


Native 


Native 


White: 2% 


Native 


Native 


Native American: 




American: 


American: 


American: 


American: 5% 


American: 0% 


Black: 0% 


American: 


American: 38% 


White: 




0% 


0% 


6% 


White: 4% 


White: 19% 


Asian: 0% 


White: 


White: 0% 


Black: 0% 




White: 34% 


White: 5% 


White: 8% 


Black: >1% 


Black: 6% 


Other: 0% 


Black: 0% 


Black: 0% 


Asian: 0% 




Black: 4% 


Black: 0% 


Black: 1% 


Asian: >1% 


Asian: 3% 




Asian: 0% 


Asian: 0% 


Other: 0% 




Asian: 0% 


Asian: 0% 


Asian: 0% 


Other: 6% 


Other: 0% 




Other: 0% 


Other: 0% 






Other: 0% 


Other: 0% 


Other: 8% 














% ELL 


82% 


26% 


38% 


9 th : 28% 


NA 


43% 


NA 


83% 


58% 


% Free/ 
reduced lunch 


65% 


100% 


90% 


71% 


68% 


100% 


NA 


81% 


55% 


Other 


All Ts are 4- 


All Ts are 4- 


6/12 Ts are 


6/16 8 th Gr. 


All Ts MSA 


All 5 th Gr. Ts 






9/12 Ts MSA 5 th & 




yr. MSA 


yr. MSA 


MSA 


MSA 


(K-5) 


MSA 






6th 



2/16 9 th Gr. 

MSA 

Note: NA refers to incomplete data sets or those data which could not be disaggregated (MSA vs. non-MSA Ss). 
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MSA mentors. MSA is guided by three mentor teachers, responsible for 
program development and implementation as well as for managing the logistics 
of the project, such as stipend payments, travel arrangements, and a host of other 
tasks. One of the mentors has been a part of the project since its inception; she 
holds a Master's degree in science education, has extensive curriculum and 
standards knowledge and expertise, and has been a teacher in Northern New 
Mexico schools for a number of years. The second MSA mentor joined the project 
at the beginning of Year 2; she has a Master's degree in education, extensive 
experience teaching in a local teaching credential program and in Northern New 
Mexico, and is currently enrolled in a Ph. D. program in education. A third 
mentor teacher joined the project in April 2003, and brings to the project 
additional experience and expertise in science and math instruction, technology, 
a Ph. D. in education, as well as extensive knowledge of and experience with the 
schools in Northern New Mexico. 

During Year 4 of MSA, as in previous years of the project, mentors served 
as project directors and program planners, and worked to develop and 
implement the goals and direction of MSA. Mentors worked extensively with 
teachers to guide their thinking and practices, and provide support to teachers as 
they implemented new strategies and approaches to teaching and collaboration. 
Each mentor observed a sub-sample of teachers at MSA sites — both returning 
teachers and teachers new to the project — an average of 6 times during school 
year. Each observation, based on a cognitive coaching model, included a pre- 
conference discussion of the lesson to be observed, the observation itself (during 
which time mentors compiled notes and used the classroom protocol developed 
in Year 2 of the project, and a subsequent debriefing session of the lesson. In 
some instances), this protocol was modified to allow mentors more time to focus 
on specific elements of the lesson and/ or to allow the mentors additional time in 
specific teaching situations, such as a science lab. Mentors also organized follow- 
up training sessions, and worked with 1 st year MSA teachers more extensively as 
needed. 

During Year 4 of MSA, mentor teachers were also involved in a number of 
community projects designed to have a long-term positive impact on the 
program. Involvement in these projects was viewed as important and critical to 
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the on-going and future success of MSA, but meant that mentors had more 
limited time and attention to devote to teachers and classroom observations. A 
fuller discussion of this issue is presented later in the report. 

Implementation of MSA Year 4 Project Goals 

The following section presents data gathered to answer the four research 
questions previously stated, with specific examples of how and in what ways 
teachers implemented various elements of the project. The primary foci of MSA 
Year 4 were to further refine strategies for instructional excellence, continue to 
foster and support collaboration amongst and between teachers and schools, 
more systematically assess student learning, and explore how and in what ways 
the cognitive coaching model could be expanded to include more teachers in the 
process of reflecting on their instruction and refining it to better support and 
promote student learning. 



Program Evolution 

To examine the multiple ways in which MSA has evolved as a model for 
professional development, a number of data gathering approaches were used. 
One such data set was created via monthly phone and e-mail contact with MSA 
mentors. The general purpose of the conversations was a check on program 
development and implementation. It also provided mentors time to reflect on the 
general status of the project, discuss recent research findings that may impact 
teachers and their practice, and plan for future staff development sessions with 
teachers based on work and interactions from the field. 

Selected MSA classrooms and after-school planning sessions were the 
focus of a site visit during the 2003-2004 school year. This site visit was designed 
to systematically collect information on the implementation of project goals. A 
modified observation protocol was used that expanded on the dimensions 
implemented by mentor teachers during their classroom observations. A sub- 
sample of classroom teachers was interviewed, either in-person or by phone, 
about their general perceptions of MSA progress, implementation of project 
goals, and specifics from observed lessons. Data collected from these classroom 
visits was used to further refine the MSA model for professional development, 
and provide information on how the project evolved. 
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There were 5 general areas of program evolution during Year 4 of MSA. 
They included: planning, collaboration, instruction, assessment and classroom 
management. Each of these areas — its goal, implementation and success — is 
described below. 

Planning. Systematic planning, combined with the identification of 
learning goals aligned with state content standards, was an area of continued 
refinement and evolution during Year 4 of MSA. In MSA, teachers work 
collaboratively to set goals and identify key concepts to teach throughout the 
year. During a previous Summer Institute, planning tools were introduced to 
facilitate the identification and implementation of important learning goals. 
Teachers worked systematically within their teams to develop a set of "key 
concepts" to teach as a mechanism for focusing student learning during the year. 
One of the many tools teachers used in developing these goals was an approach 
called "data not guesswork" (DNG), a system that allows teachers to monitor 
student progress towards targeted learning goals. Teachers had the opportunity 
to use a variety of instructional planning tools for Year 4, including the 
development of DNG questions and common instructional goals and 
assessments, as they worked to develop year-long instructional plans for their 
classes. 

Results from surveys, interviews, and observations, indicated that overall, 
most teachers were regularly and consistently planning instructional programs 
that focused on student learning. Table 4 presents results from the survey. 
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Table 4 

Program Planning: Curriculum and Articulation 

Please respond to the following based on your 2003 - 2004 

implementation of MSA ideas. N=43 

Mean 

(SD) 



a) I develop year-long and short-term goals for 3.9 

my students. (0.9) 

b) I select content to meet the learning goals of 4.5 

my students. (0.6) 

c) I design and adapt curricula to meet the 4.3 

needs of my students. (0.7) 

d) I use instructional strategies that develop 4.3 

and promote student understanding. (0.8) 

e) I work with my colleagues within disciplines 3.8 

to set goals and standards for learning and (1.2) 

achievement. 

f) I work with my colleagues across disciplines 3.7 

(content areas) to set goals and standards for (0.9) 

learning and achievement. 

g) I developed goals for "data not guesswork" 3.4 

and used them to guide instruction. (1.2) 



Note. Scale: l=never, 3= sometimes, 5=always, NA=Not applicable. 



As Table 4 indicates, during 2003-2004 in general, teachers reported 
planning instruction in meaningful ways, including working with colleagues to 
develop long-term learning goals for students. Disaggregation of the responses 
revealed that 3 rd and 4 th year MSA teachers reported engaging in planning 
activities more frequently than did newer MSA members, suggesting perhaps 
that establishing goals and planning with colleagues becomes a more regular 
feature of teachers' approaches to instruction as they gain experience with MSA 
and its goals. Additionally, most teachers indicated that their instructional 
planning, articulation, and collaboration with their colleagues had changed "a 
great deal" on a 5-point rating scale due to their participation in MSA. 

MSA teachers made the following comments about the role that 
instructional planning and articulation of goals and expectations had on teaching 
and learning in their classrooms: 



12 




Teacher 1 

This year we met as part of MSA and it helped me to focus on what I 
needed to do to improve my teaching and to better assist my students 
with their learning. 

Teacher 2 

Before MSA I would plan with one or two teachers on occasion. Since 
being part of MSA I have been part of a school reform toward 
departmentalization and on a planning team with all 6 sixth grade 
teachers at my site. This planning has really helped to strengthen my 
instruction. 

Teacher 3 

I am using short-term and long-term goals in my teaching and for my 
students. This has helped me stay focused on what I am teaching. 
Students are writing and signing mission/vision statements. These ideas 
help tremendously in class. 



Collaboration. A second important feature of MSA that reflects program 
evolution is the area of professional collaboration. Three general types of 
professional collaboration are supported by MSA: (a) within-school 

collaboration, where teachers at the same school meet and collaborate about 
instruction and MSA during and after school; (b) across school collaboration, 
where teachers at the same grade level and/ or content area collaborate at MSA 
Summer Institutes and MSA meetings throughout the school year; and (c) 
general MSA project collaboration, where teams and teachers participate in joint 
learning experiences and opportunities. Each of these collaborations provides 
teachers with different kinds of opportunities for professional growth and 
interaction, with the ultimate goal of increasing student learning. 

Interviews, observations, and survey data indicate that increased 
collaboration of all types was generally a positive experience for MSA teachers. 
Teachers reported that greater regular collaboration with their peers served to 
strengthen their sense of belonging to the MSA team at their sites, to build on 
their sense of membership in the MSA intellectual community, and to deepen 
their understanding of how to implement MSA tools, strategies, and ideas more 
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effectively. Comments below from teachers illustrate the general findings about 
the effects of collaboration on teachers during Year 4 of MSA. 

Teacher 1 

We have all been working and talking more with each other. This was not 
happening before MSA. Many of us were divided on our teaching and 
beliefs, but with MSA we have come to realize we all need each other. 

Teacher 35 

Before joining MSA, I would close my classroom door and keep 
everything I did inside. There was never any collaboration between grade 
level teachers or any other teachers. It is so much nicer and more 
productive for our students to share ideas and work out problems with 
colleagues. 

Teacher 41 

Our level of articulation and collaboration has increased so much because 
of MSA. Through our conversations we can identify problems w/ teaching 
strategies and pose probable solutions and alternatives. 

Teacher 29 

We now work well now as a sole school and also as a group with other 
districts. 

Teacher 8 

MSA has provided our staff the time and place to share, plan and 
collaborate with each other. Spending time together during the summer 
has also strengthened our bonds in a personal and professional way, 
which can only improve our working relationships. 



Instruction. As the Math and Science Academy has evolved as a program, 
so too have project members' ideas and understandings of what quality, 
standards-based instruction means and looks like in the classroom. The MSA 
model for quality teaching incorporates a wide-variety of research-based 
approaches to learning and instruction. Central to the project is the notion that 
teachers move from a didactic, input-only approach to teaching (teacher lectures, 
students learn) to one that affords students' multiple ways and multiple 
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opportunities from which to learn concepts, ideas and information. The 
introduction of "data not guesswork", presented and discussed in detail in later 
paragraphs, is one instructional approach implemented during Year 4 of MSA. 
The use of graphic organizers to support student learning is another instructional 
strategy supported by MSA. Systematically organized, visual representations of 
concepts are the key elements of the use of graphic organizers. 

The first year these strategies were introduced, a number of middle school 
teachers reported reluctance to use tools they viewed as "too elementary, and too 
babyish" for their middle school students. The introduction of graphic organizers 
and "data not guesswork" into elementary classrooms occurred more easily and 
more seamlessly during Year 4 than in previous years, in part because of the 
tradition of incorporating art and other whole group record keeping systems into 
the curriculum at the elementary level. Another factor that helped to increase 
implementation was support for the ideas from other MSA teachers who had 
tried the instructional approaches in their classrooms and were successful in 
their implementation. Across grade levels and content areas — from kindergarten 
to ninth grade — in math, science, social studies, and language arts, teachers 
reported a positive experience when new instructional tools were introduced and 
used, as well as reported increases in student learning and student effort. 

A companion element to the instructional strategies advocated and 
embraced by MSA is the use of a variety of instructional materials to support 
student learning. Table 5 displays information on the nature and types of 
instructional materials and resources MSA teachers reported using during the 
2003 - 2004 school year. 
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Table 5 

Instructional Materials and Resources 



Please respond to the following based on your 2003 - 2004 

implementation of MSA ideas. N=43 

Mean 

(SD) 



a) I structure learning time to allow students to 4.0 

engage in projects and/or investigations. (0.7) 

b) I create a setting for student work that is 4.3 

flexible and supportive of student learning. (0-7) 

c) I ensure a safe learning environment. 4.8 

(0.5) 

d) I make available tools & materials to 4.4 

students to support learning. (0.7) 

e) I make available print resources to students 4.2 

to support learning. (0.8) 

f) I make available technological resources to 3.7 

students to support learning. (0.8) 

g) I use graphic organizers to support learning. 4.2 

(0.9) 

h) I engage student in designing their learning 3.2 

assignments. (0.9) 



Note. Scale: l=never, 3=sometimes, 5=always, NA=Not applicable. 



Ensuring a safe learning environment appears to be a frequently reported 
key element in establishing productive learning situations for MSA teachers. 
Other tool use is more varied: teachers reported that technology use, in particular 
at the elementary level, is more limited. As schools in New Mexico move more 
technology into elementary classrooms and curriculum, it is important for MSA 
teachers to continue to learn to use technology as a resource to support student 
learning. 

For 3 rd and 4 th year MSA members, technology use has increased 
dramatically over the past 3 years. Veteran teachers who initially expressed 
serious doubt about their capacity to use technology in meaningful ways are now 
using computers for instructional planning, record keeping, communication with 
parents and other teachers, and grading. A number of veteran teachers have 
embraced technology use in their classrooms; students regularly use technology 
and a variety of programs to support and enhance their learning. 
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Comments from teachers about changes in their instructional approaches 
as a result of MSA participation include the following ideas: 

Teacher 37 

Before MSA, my teaching was textbook guided. Now I use standards to 
drive my teaching and I have daily learning goals. My students 
understand standards and learning goals. 

Teacher 13 

I now have a broader view of the importance of student learning. I also 
have many tools and guides to use. I know I will reach out and touch 
more students more positively in the classroom. 

Teacher 9 

One of the biggest changes in my instruction has come by focusing on 
student learning and adjusting my teaching to students needs. I've re- 
visited my cooperative learning strategies and refined them in use. I'm 
more aware of the standards and benchmarks and the role of teaching 
about the context. 



Assessment. As MSA teachers have worked to implement new and varied 
instructional strategies, and provide standards-based instruction, a parallel focus 
has arisen in the area of assessment. Teachers are recognizing the need for 
common assessments across grade levels and in specific content areas. MSA has 
addressed teachers' needs with respect to assessment by providing teachers 
access to various resources, in particular those that focus on the use of rubrics or 
matrices that specify student performance levels. 

Table 6 shows teacher ratings of the frequency with which they use 
various strategies to assess student learning. Overall, teachers reported using the 
assessment strategies advocated and supported by MSA "sometimes." Research 
on teacher assessment practices confirms the difficulty and challenge teachers 
face in developing and implementing assessment strategies that are feasible and 
timely, and that provide reliable and accurate information about student 
learning. 
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Table 6 

Assessing student learning 

Please indicate your observations regarding 2003 - 2004 

student learning and achievement this year. N=43 

Mean 

(SD) 



a) I systematically gather data on my students 3.5 

and their learning in my classes. (1.0) 

b) I analyze assessment data on a regular and 3.6 

timely basis to inform and guide my (1.0) 

teaching. 

c) I guide my students in self-assessment. 3.2 

(0.9) 

d) I use student data, observations of teaching, 3.6 

and interactions with colleagues to reflect on (0.8) 

and improve my teaching practices. 

e) I provide students with information on how 3.9 

their work will be assessed. (0.9) 

f) I provide students with examples and 3.7 

models of what represents "good work." (0.8) 

g) I use a wide variety of assessments to help 3.7 

me understand students' ideas and learning. (0.8) 



Note. Scale: l=never, 3=sometimes, 5=always, NA=Not applicable. 



A number of teachers noted the impact that MSA is having on their 
assessment practices, indicating an increase in their knowledge and 
understanding of assessment. Teacher comments about assessment included the 
following: 

Teacher 1 

I am learning from MSA that assessment is far more than standardized 
tests. Assessment can be formal and informal. I am also learning how to 
use rubrics to assess student learning and how to help my students 
develop portfolios. 

Teacher 15 

MSA has made me more aware of assessing student work before and after 
a skill has been taught. It has helped me to know whether or not the 
students have achieved the learning that was targeted in the instruction. 



18 




The assessment lets me know what the end result of the learning looks 
like. 



Teacher 21 

I have learned so many different ways that I can use to assess my students 
work, which will not only benefit my students but also myself. They will 
have more responsibility for their own learning and assessing themselves 
as well as help me to keep up with their progress. 

Teacher 38 

I am much more knowledgeable as to the different types of assessments 
which can be used. I work hard to determine which instrument will 
provide me with the data I am seeking. Also, my students are assessed in 
multiple ways. 



A number of teachers did acknowledge however, the important and on- 
going need for further developing and refining their assessment techniques, 
strategies and understandings. These views were reflected in informal interviews 
conducted at the 2004 Summer Institute. There, teachers commented "the more 
we learn about assessment, the more we understand how much we don't know." 
Additional comments from new MSA members regarding assessment included: 

Teacher 17 

As a result of my first year in MSA I became more aware of my 
assessment practices. I know now that I need to improve in this area and 
intend to make this my focus for the coming year. 



Classroom management. The development and delivery of quality 
instruction, quality assessments, and quality tools used by quality teachers to 
support and strengthen student learning are the long-term goals of MSA. As 
MSA teachers and mentors have come to realize however, to increase and 
maximize student learning, classrooms must have protocols and structures in 
place that support productive teacher and student interactions. During the 
Summer Institute of 2003, a number of approaches to classroom management 
were presented, including strategies from Harry Wong (e.g., Wong & Wong, 
1998). The essential message of this approach is that routines and protocols must 
be established and learned early in the school year, and reinforced throughout 
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the year, to allow all students the opportunity to develop their ideas and 
understandings and to learn effectively. Routines help students understand the 
rules and expectations in a classroom, and allow the teacher to focus on teaching, 
rather than discipline. MSA mentors selected this research-based approach to 
classroom management after observations in numerous classrooms revealed the 
challenges many teachers faced with management issues. 

MSA teachers reported great success in learning and implementing 
classroom management strategies from the resources provided by MSA. It is 
interesting to note that, even for veteran teachers, the opportunity to learn new 
classroom management approaches was important and meaningful. Teacher 
reflections on classroom management approaches are featured below. 

Teacher 9 

After last summer and learning Harry Wong - my classroom management 
was so much smoother. Management strategies learned along the way 
helped, but those strategies really pulled everything together for me. 

Teacher 41 

I have become aware of providing an environment, which promotes self- 
directedness, as well as a cooperative environment. From Dinah Zikes, I 
have set up a student work center where students can access graphic 
organizer materials. I have also implemented Harry Wong's strategies on 
classroom management. These have proved most effective, in particular 
the establishment of routines and procedures. These strategies have made 
me a more effective teacher. 

Cognitive Coaching: Development and Implementation 

During the past four years of the Math and Science Academy project, a 
cornerstone of the program has been an approach to implementing instructional 
change and reform known as "cognitive coaching" (Costa & Garmston, 1998). 
The research-based theory underlying this approach is the notion that teacher 
change and development is most likely to occur in the context of guided 
reflection, where conversations about teaching practices, teacher-student 
interactions and instructional effectiveness, are continual, on-going and used to 
modify and improve teaching. MSA mentors have appropriated the cognitive 
coaching protocol from the cognitive model, and revised it over the past 4 years 
to more closely reflect MSA goals and objectives. Mentors attribute much of the 
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MSA project success to the careful and thoughtful interactions that result from 
the coaching experience. 

The cognitive coaching approach is intensive and time-consuming for 
both participants. Prior to the instructional observation a pre-conference 
conversation occurs, requiring the teacher and the mentor to focus on the 
intended goals and objectives for the lesson and the observation. The classroom 
observation requires a minimum of 45 minutes of time, and often ranges to one 
and half hours, during which time the coach/ observer records careful notes 
about the lesson to be shared with the teacher regarding the agreed upon area of 
focus. This is followed by a debriefing/ post-instructional conference, which 
ranges in time from approximately 45 minutes to 3 hours. The purpose of the 
debrief is to allow the teacher and the observer an opportunity to interact and 
reflect about the lesson, its successes and challenges. In total, the coaching 
"package" requires a commitment of approximately 5 total hours of time from 
each participant. The challenge for MSA during Year 4, a year of expansion and 
scaling up, was, as one mentor phrased it: "to keep the integrity of the MSA and 
the coaching sessions, and make it work for 72 teachers at nine different 
schools!" 

To accomplish the scale-up implementation and expansion for Year 4 of 
MSA, third and fourth year MSA teachers were invited to take on coaching roles 
and responsibilities. Mentors continued to meet and observe in coaches' 
classrooms with all of the 1 st year participants, as well as to continue more 
limited observations in veteran MSA classrooms. Mentors worked extensively 
with teachers at four new elementary schools, those without veteran MSA 
members. Table 7 below displays the schedule and responsibilities for coaching 
at each of the sites. 
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Table 7 



Year 4 MSA Cognitive Coaching Schedule 



Schools 


Coaches 


Average Number 
of Coaching 
Session 


Coaching Description 


Middle School A 


• 2 fourth year 


6 total, 3 per 


Flexibility in scheduling 


and Elementary 
School E 


MSA members 
• MSA mentors 


semester 


allowed Cs and Ts to easily 
observe lessons throughout 
day. Debriefing occurred at the 
conclusion of the day. 


Middle School B 


• 2 fourth year 


6 total, 3 per 


Continued coaching support 


and Elementary 
School F 


MSA members 
• MSA mentors 


semester 


from MSA mentors. Coaching 
schedule challenging due to 
limited release time for Cs, and 
conflicting after-school 
meetings. 


Middle School C 


• MSA mentors 


6 total, 3 per 
semester 


Mentor observed in classrooms. 
Conflicting agendas and school 
politics made consistent 
coaching schedule challenging. 


Middle School D 


• 2 third year 
mentors 


6 total, 3 per 
semester 


Changing staff, new additions 
to MSA cohort presented 
challenges to coaching model. 
Productive after-school 
meetings. 


Elementary School 

P 


See above 


See School A above 


See above 


C 

Elementary School 


See above 


See School B above 


See above 


r 

Elementary School 
G 


• MSA mentors 


6 total 


Mentor observed in classrooms, 
provided demonstration 
lessons. Initially led after-school 
meetings, responsibility 
gradually assumed by MSA 
teachers. 


Elementary School 
H 


• MSA mentors 


6 total 


Mentor observed in classrooms, 
led demonstration lessons. 
Mentor participated in after- 
school meetings. 


Elementary School 

I 


• MSA mentors 


6 total 


Mentor observed in classrooms, 
led demonstration lessons. 
Mentor participated in after- 
school meetings. 
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Cognitive Coaching Impact on Teachers 

MSA teachers were overwhelmingly positive in their reactions to the 
coaching experience they experienced during the 2003-2004 school year. 
Teachers reported that coaching functioned in a variety of ways to support their 
learning and development as instructional leaders. Many teachers, who were 
initially apprehensive about being "observed" by MSA mentors, reported that 
they viewed the sessions as non-threatening, productive, and as an important 
tool to facilitate the reflection process. Teachers reported the coaching experience 
as productive, interesting, and important in the development of their 
understandings of how and in what ways to refine their teaching. Comments 
from teachers regarding the coaching experience are below. 

Teacher 9 

The coaching experience helped me to really focus on my processes of 
teaching. Isolating specifics about lesson goals, reteaching, etc. made me 
analyze, adjust and better my skills — and I didn't feel threatened by my 
coaches like I initially thought I would. When my coach and I did our pre 
and post conferences, I surprised myself that I knew my goals, etc. — my 
responses were right on. How validating as a teacher to have a real 
discussion about my teaching! 

Teacher 22 

The coaching experience allowed me to reflect on my teaching methods. 
The feedback offered by this experience brought forth images that I wasn't 
aware existed. For example, I don't always remember to state my goals up 
front; I need to break up the lessons into digestible pieces. 

Teacher 33 

Positive feedback and constructive criticism are wonderful for a relatively 
new teacher like me. I need to know what I'm doing right and where I 
need to improve. 

Science is my weakest area. The coaches really helped me by giving me a 
clearer picture of standards and how to deliver them to the students. 

Teacher 18 

The cognitive coaching sessions were very useful for me in that they made 
me more conscientious of planning for instruction in an effective manner. 
I always viewed lesson planning in a traditional "mode," whereas, with 
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the MSA cognitive coaching, various elements of instructional planning 
were much more effective in impacting student learning. 

In one district, the elementary school was placed on "corrective action" by 
the state due to declining test scores. As a school and district, the decision was 
made to include elementary teachers in the 4 th year MSA cohort and have the 
MSA middle school teachers function primarily as their coaches. Initially, there 
was a great deal of concern about this approach— veteran MSA teachers 
questioned whether they would be able to implement the processes and 
procedures critical to the cognitive coaching model. MSA 4-year veterans 
reported that prior to MSA Year 4, it was easy to "point fingers, and blame 
elementary school teachers for the ways (or not) in which they were preparing 
students for middle school." Other concerns focused on the challenges of 
scheduling observations and debriefs, and they could be arranged in a feasible, 
workable manner. Elementary teachers too wondered about the nature of the 
relationship between their teaching staff and the middle school staff. 

The result of this decision to include the elementary teachers in the MSA 
cohort was interesting. Overall, teachers, coaches and administrators were very 
positive in their response to MSA "scale up implementation" at this site. At an 
after school meeting in May, the majority of participants reported that: "MSA has 
made a huge difference in what we think about teaching and learning at our 
school. We are learning together, trying to do cognitive coaching together and 
communicating much more effectively as a group, and as a learning 
community." Participants noted the importance of matching coaches with 
teachers in terms of background and experience to ensure a common vocabulary 
and understanding, as well the importance of similar philosophies in terms of the 
application and implementation of certain instructional approaches with specific 
sub-populations of students. These concerns were discussed openly, and 
suggestions and comments on how to solve the situation in future years were 
elicited. 

Additional comments from MSA teachers about the ways in which the 
coaching experience impacted thinking, teaching, and learning are listed below. 
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Teacher 27 

Specifically, the coaching experience allowed me to identify with some 
problems I encountered during the lesson & helped me to modify the 
lesson in order to reach desired outcomes. For example, I used to pose the 
class w/ a probing question to large groups. Many were not even engaged 
. . .From this I learned to implement "really round the table," a Kagan 
structure, in which all students were engaged. 

Teacher 1 

The pre and post conferences helped me assess with an open mind and 
the coach helped in being objective. Also, I enjoyed as well as my students 
the help and involvement of the coaches during their time in the 
classroom. When my "mentor" came into the classroom while I was 
presenting a math concept dealing with circumference, radius, and parts 
of a whole. She was able to demonstrate one of the concepts more clearly 
than I was able to. I greatly appreciated this help as well as did the 
students. This approach (coaching) is very useful; most often the principal 
does not get into the classroom often enough to help give feedback to 
teachers on how they are doing. This provides a great opportunity for 
teachers to get that feedback that they need. 

MSA Impact on Coaches 

Coaches also reported an impact on their thinking and understanding of 
MSA goals and objectives as a result of participating in the coaching process. 
Coaches, in general, were experienced teachers with a minimum of 3 years of 
participation in MSA. As a result, teacher-coaches were in the unique position of 
having experiencing the coaching protocol from a "coachee" as well as the 
"coach" role. Coaches found the experience of visiting other classrooms, 
observing instruction and then discussing the observations, to be important and 
meaningful in a number of different ways. First, the experience helped many 
coaches to "stop the blame game," that is, to minimize their perceptions that the 
reason for poor student performance was because of shortcomings of a previous 
teacher, grade level, or content area. Coaches experienced first-hand the ways in 
which student ideas developed (or didn't) throughout the grades. Second, 
coaches reported that the experience helped them to reflect on their own teaching 
experience in ways that did not occur in other settings. One coach wrote: 



The coaching experience has really brought to light a number of different 
ideas I have about teaching and learning. I always thought a lesson had to 
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be structured a certain way to maximize learning, but after visiting other 
classrooms, I can see the power in other approaches. Something else that 
really struck me is how interconnected we all are in the learning 
community. I saw science concepts introduced in 3 rd grade that we revisit 
again in 8 th grade science. Coaching has highlighted the importance of 
clear articulation of learning goals. It has also helped me to develop more 
sophisticated questioning strategies. 



One of the challenges encountered by MSA mentors and coaches was the 
addition of new teachers at a school site during the middle of the year, due in 
large part to staffing changes. New teachers represented a welcome addition to 
the project, but coaches often found it difficult to inculcate teachers to MSA 
concepts and ideas absent a formal teaching/ learning situation such as the 
Summer Institute. Thus, the notion of a common learning experience, and its 
importance in developing a common language and set of goals like those 
provided at the Summer Institute, was highlighted by teacher and coach 
experiences in Year 4 of MSA. 

MSA Impact on Teaching and Learning 
Student Achievement 

Data presented in this section on project impact on students are culled 
from a number of different sources: standardized achievement test scores and 
teacher and administrator reports. As MSA teachers have become more familiar 
with the project, and have implemented project goals to a greater degree, 
positive changes in students' performance and achievement have emerged in 
some MSA classes and schools. 

Test data reported here were made available to UCLA/ CRESST through a 
third party research group to ensure student confidentiality. Each student was 
assigned a unique ID; codes and data for students will be maintained for the 
duration of the MSA project and then destroyed. Additional tables with more 
detailed information on student performance can be found in Appendices. 

In this evaluation report, test scores are again reported using a scale score 
(SS). Scale scores are based on a scale of 1 - 1000, with equal intervals between 
each number on the scale. The scale can be applied to all groups taking a given 
test, regardless of group characteristics or time of year, making it possible to 
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compare scores from different groups. Scale scores are used to facilitate 
appropriate comparisons and analyses for various statistical purposes; for 
example, scores can be added, subtracted, and averaged across test levels. These 
computations make direct comparisons among students and their scores 
possible. Further, scale scores made it possible to compare individual scores to 
groups, and to compare an individual's pre-test and post-test scores in ways that 
are statistically valid. These types of comparisons cannot be made with percentile 
rankings or by using grade-level equivalents. Note: School pseudonyms used 
below correspond to those listed in Table 3 (page xx). 

Standardized test scores. Test scores reported in this section are from the 
2003-2004 school year for MSA schools. The CTB, published by McGraw Hill, 
was administered at the end of March 2004. Below are tables that present 
information from various sites, for various subjects, along with brief analyses of 
the scores. For the 2003-2004 academic year, norm-referenced test data were not 
available for 4 th or 8 th grade MSA students in New Mexico due to a change in 
assessments (4 th and 8 th grade students took a standards-based assessment, also 
published by McGraw Hill). Those data were not available for this report. 

Data tables are presented for some MSA schools as a matched data set, to 
allow comparisons of students from one year to the next. The range for 
percentages of matched student scores varies from school to school. At some 
sites, the matched data set represents approximately 60% of the cohort. At other 
schools, the percentage of matches is greater (approximately 90%). These 
matched data sets provide information about MSA's impact from year-to-year on 
student achievement. Thus, the total number of students in these tables varies 
from Table 3 (see page 11) due to changes in student populations and availability 
of data. 

Data are also presented in this section for School C; it was not possible to 
disaggregate the data and examine changes in a cohort's scores. In this case, data 
are presented for ah 7 th grade students since the inception of the project. 
Additional tables of student data from year-to-year scores for a specific grade 
level can be found in the Appendices. Again, caution should be used in 
interpreting the scores for these sections, due to small sample size and changes in 
the student population from year to year. 
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Table 8: Matched Cohort Data 
School A (n=18) 





Reading/ 

SS 


Language 
Arts/S S 


Math/SS 


Science/SS 


Social 
Studies/S S 


2003 
6 th grade 
(MSA) 


655 


655 


650 


650 


650 


2004 
7 th grade 
(MSA) 


654 


661 


644 


657 


663* 


Change 


-1 


+6 


-6 


+7 


+13* 



Note: *p < .05 



Seventh-grade student performance at School A reflects a slight, but not 
statistically significant, increase in scores in language arts and science. Positive 
increases in social studies were statistically significant at the p < .05 level. 
Reading and math scores showed a slight, but not statistically significant 
decrease. At School A, all teachers are MSA teachers, and middle school teachers 
are 4-year members of MSA. Cohort size at School A is small (n < 20 students) 
and as such, caution should be used to interpret scores. 



Table 9: Matched Cohort Data 
School B (n=47) 





Reading/ 

SS 


Language 
Arts/S S 


Math/SS 


Science/SS 


Social 
Studies/S S 


2003 
6 th grade 
(MSA) 


647 


655 


646 


647 


649 


2004 
7 th grade 
(MSA) 


661* 


665 


662* 


664* 


663* 


Change 


+14 


+10 


+16 


+17 


+14 



Note: *p < .05 



At School B, 2003-2004 student performance in 7 th grade showed 
improvement in all content areas, with statistically significant changes from the 
2002-2003 school year, in reading, math, science and social studies. Language 
arts scores increased slightly but were not statistically significant. The majority of 
teachers at School B participate in MSA and are 4-year MSA veterans. 
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Table 10 

School C: All 7 th Grade (including non-MSA students) 
MSA CTB/Scale Scores 





Reading/S 

S 


Language 
Arts/S S 


Math/SS 


Science/SS 


Social 
Studies/S S 


Year 












2000 


635 


637 


616 


625 


631 


2001 


642 


644 


635 


639 


643 


2002 


648 


648 


641 


642 


643 


2003 


653 


651 


651 


656 


650 


2004 


645 


645 


644 


648 


648 


Score 

change 

'03 - '04 


-8 


-6 


-7 


-8 


-2 


Score 

change 

'00 - '04 


+10 


+8 


+28* 


+22* 


+17* 



Note: *p < .05 



At School C, student scores showed a slight decline in all content areas 
from 2003. Since the inception of MSA in 2000, however, all content area scores 
have increased; changes in math, science, and social studies are statistically 
significant at the p < .05 level. 

It should be noted that at School C, only seven members of the 20-person 
faculty were MSA participants; one "team" of teachers that includes math, 
science, language arts and social studies teachers, along with three other 
teachers. At School C then, the majority of students have classes taught by non- 
MSA teachers. In Table 8, scale scores reported are for all students in all subjects 
at the school, even in cases where students were taught by non-MSA teachers. 
Because of cohort overlaps (some students had MSA teachers for some but not all 
subject areas) and varied testing conditions, it was not possible to disaggregate 
student scores by MSA vs. non-MSA teacher participation. Comparison of project 
impact on MSA vs. non-MSA teachers is therefore limited. 
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Table 11 (Espanola 9) 

School D: 9 th Grade: MSA CTB/Scale Scores 





Reading/SS 


Language 
Arts/S S 


Math/SS 


Science/SS 


Social 
Studies/S S 


Year 












2003 


658 


653 


662 


663 


653 


2004 


665 


653 


669 


674 


661 


Score 

change 

'03 to '04 


+7 


0 


+7 


+11 


+8 



Note: *p < .05 



Data for School D are limited in scope: CRTs (criterion referenced tests) 
were administered to 8 th -grade students, thus making comparisons to previous 
8 th -grade to 8 th -grade scores invalid. However, there are promising gains in 
student scores from the 2003 to the 2004 school year for 9 th -grade students, in 
particular in the areas of reading, math, and science. 

Schools E - 1 

Data for MSA schools E - I are presented in Table 12; this is the first year 
of MSA at these sites, and data are intended to capture baseline information on 
student achievement. Previous evaluations have noted the challenge of 
expectations for score improvement after the a new program or instructional 
approach is introduced, and this year, modest increases in 1st year scores were 
observed in some content areas at some schools. Of particular interest are the 
changes in math and science scores, which remain a focus for MSA, and indeed is 
where many elementary teachers perceive the greatest need for additional 
knowledge, strategies, and support in their instruction. Scores are reported for 
2003 and 2004 fifth grade students only, due to data availability. 
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Table 12 

Data for MSA Schools E-I 



MSA 5 th 
Grade Elem. 
Schools 


Reading/SS 


Language 
Arts/S S 


Math/SS 


Science/SS 


Social 
Studies/S S 


School E 
2003/2004 


664/659 


662/664 


638/627 


659/646 


648/650 


Change 


-5 


+2 


-9 


-7 


+2 


School F 
2003/2004 


641/637 


638/634 


631/627 


636/636 


636/636 


Change 


-4 


-4 


-4 


0 


0 


School G 
2003/2004 


647/631* 


647/638* 


628/622 


638/636 


634/632 


Change 


-16* 


-11* 


-6 


-2 


-2 


School H 
2003/2004 


642/640 


642/648 


623/639* 


637/640 


630/638 


Change 


-2 


+6 


+16* 


+3 


+8 


School I 
2003/2004 


647/648 


652/652 


627/636 


632/643* 


632/639 


Change 


+1 


0 


+9 


+11* 


+7 



Note: *p < .05 



Results at the elementary school were varied. At some sites, scores 
showed modest but generally not statistically significant improvement. At other 
sites, scores remained the same or decreased slightly. As noted in previous 
evaluation reports, it is challenging for teachers to implement new ideas and 
approaches consistently when they are learning new approaches. 

Additional data were available for elementary students at some sites. 
Below, in Tables 13 and 14 are matched cohort data from 5th grade MSA schools 
E and F. Statistically significant growth occurred in reading and language arts, 
while math and social studies showed slight improvements. Science scores at 
School E declined slightly for the 2003- 2004 school year, but the decrease was 
not statistically significant. 
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At School F (Table 14), the matched cohort scores increased in all content 
areas (reading, language arts, math, science, and social studies). These changes 
were statistically significant at the p < .05 level. 



Table 13: Matched Cohort Data 
School E (n=18) 





Reading/SS 


Language 


Math/SS 


Science/SS 


Social 






Arts/S S 






Studies/S S 


2003 

4th grade 

(pre- 

MSA) 


638 


645 


622 


648 


645 


2004 


658* 


665* 


627 


644 


648 


5th grade 












MSA 












Change 


+20* 


+20* 


+5 


-4 


+3 


Note: *p < .05 












Table 14: Matched Cohort Data 








School F (n=41) 












Reading/SS 


Language 


Math/SS 


Science/SS 


Social 






Arts/S S 






Studies/S S 


2003 
4 th grade 

(pre- 

MSA) 


617 


626 


604 


620 


621 


2004 


641* 


639* 


629* 


638* 


638* 


5 th grade 












MSA 












Change 


+24* 


+13* 


+25* 


+18* 


+17* 



Note: *p < .05 



Teacher Reports of Student Learning 

In addition to standardized achievement test scores, teachers were asked 
about their impressions of project impact on student learning and achievement. 
Teachers reported overwhelmingly that student learning was positively 
influenced by their own (teacher) participation in MSA. Fewer teachers reported 
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a positive impact on students from their use of DNG ("data not guesswork"), 
perhaps a function of the fact that only 60% of teachers reported implementing 
the strategy effectively in their classrooms during the 2003-2004 school year. 
Table 15 below displays teacher ratings of MSA impact on student learning and 
achievement. 

Table 15 

Teacher observations of student learning and achievement 

Please indicate your observations regarding 2003 - 2004 

student learning and achievement this year. N=43 

Mean 



(SD) 



a) 


I have observed changes in student learning 


3.8 




and achievement this year. 


(0.9) 


b) 


My participation in MSA had a positive 


4.7 




impact on my students' learning and 
achievement this year. 


(1.2) 


c) 


MSA helped increase student learning and 


4.2 




achievement. 


(0.6) 


d) 


Students learned more because of 


4.1 




cooperative learning opportunities in my 
classroom. 


(0.9) 


e) 


Students benefited from the use of DNG in 


3.6 




my classroom. 


(1.2) 



Note. Scale: l=disagree, 3=moderately agree, 5=strongly agree, NA=Not applicable. 



Observations and interviews also suggest that students benefited from 
their teachers' participation in MSA. In many MSA classrooms, students 
appeared to understand learning goals; procedures and protocols were well 
established and understood, and a wide variety of instructional activities were 
observed. Many teachers reported their perceptions that careful planning and the 
more regular use of assessment to check on learning goals also contributed to 
increases in student learning. Another source of improvement of student 
achievement reported by teachers was the clear connection to standards, and its 
influence on providing standards based instruction. 
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Teacher 1 

I saw a lot of growth w/ my students this year, especially in math. I used 
the concept statement to direct my instruction. 

Teacher 2 

Cooperative learning, when used properly, is a WONDERFUL thing. 
DNG has worked wonders in my classroom. My students really 
benefited. 

Teacher 3 

One MSA idea that helped increase student learning in my class was the 
problem of the day. Beginning class using a multiple-choice question to 
practice test-taking strategies (DNG) helped my students with reading- 
learning vocabulary, learning to eliminate answers and solving problems. 



MSA Impact on Administrators 

Another important area of MSA's program impact related to school 
principals. MSA principals were interviewed to gather information on their 
understandings and views of the impact of MSA on teachers, students, and the 
overall learning environment at their schools. Of the nine MSA principals, six 
were new to their schools and to MSA; the other three principals were familiar 
with the project and understood MSA goals and objectives. 

Overall, MSA principals were positive in their views about the project and 
its implementation. Administrators were impressed by the increased levels of 
collaboration observed at their sites, by the evidence of careful instructional 
planning and implementation of different instructional strategies, and by the 
consistency in approaches to classroom management. Commented one principal: 

I’ve seen a lot of different projects in my time as an administrator, but this one 
(MSA) seems to be different somehow. Teachers are working more together as 
teams. In fact, their excitement about MSA is influencing other non-MSA 
teachers. Everyone wants to be part of the project and teach more effectively. 



Principals also commented on the power and cohesion of the MSA teams. 
They saw school-site leadership developing as the school year progressed, and 
reported that the ideas and reflections on teaching practices, whiled guided by 
MSA mentors, also illustrated a great deal of teacher ownership. This sense of 
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ownership helped imbue teachers with a sense of responsibility to the project 
and to working together as a team. 

One principal reflected on the expansion of MSA at his site, from 4 
teachers to 12 teachers. He said: 



The introduction of MSA this year has gone really well this year. I’m seeing a big 
increase in time on task for students, a big increase in time-spent learning. We 
still need to focus on learning time. Recess, field trips, all of that is important, but 
it is instructional time that matter most. I’m seeing great progress in all of the 
classrooms at our school . . . quality classroom management is increasing, and is 
more effective. Our elementary school has really unified in terms of discipline, in 
terms of what the expectations are. Those decisions have been made as a group at 
our Monday meetings. It’s an attitude of “let’s do this, we think it will improve 
student learning. . .” That’s how I’ve seen MSA grow and develop at our site this 
year. 

Program Effectiveness 

In general, teachers, students, administrators and mentors were positive 
about Year 4 of MSA. Classroom observations, interviews, and survey results 
provide data to support these conclusions. Teachers were asked to rate MSA's 
overall effectiveness in a number of areas. These survey results are displayed in 
Table 16 below: 
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Table 16 

MSA Program Effectiveness 



Overall 
2003-2004 
N = 43 



How effective was MSA in the following areas? 
wjjj 

a) Familiarizing you with standards-based instruction 4.6 

( 0 . 6 ) 

b) Developing your knowledge of state frameworks for content 4.4 

areas (0.7) 

c) Helping you develop interdisciplinary curriculum units 3.8 

( 1 . 1 ) 

d) Providing demonstration lessons that were meaningful and 3.9 

relevant to you and your students (0.8) 

e) Sharing assessment strategies 4.1 

(0.9) 

f) Helping you to develop rubrics to support instruction 3.8 

( 1 . 0 ) 

g) Informing/involving the community about MSA goals and 3.8 

objectives (0.9) 

h) Helping you understand how to use technology effectively 4.0 

(• 9 ) 

i) Assisting you in implementing cooperative learning activities 4.4 

(07) 



Note: Scale-l=Not effective; 3=Somewhat Effective; 5=Highly Effective. 



Teachers were also asked to rate their own knowledge and skills as a 
result of their participation in MSA. This self-assessment reflects the approaches 
and strategies most critical to successful MSA implementation. Teacher ratings 
of their knowledge of content area and standards were strong, as was their rating 
of content standard knowledge. Teacher confidence in teaching their content area 
was also strong, although a number of elementary teachers made mention of 
their need for additional mathematical content knowledge. Teachers rated their 
knowledge of technology skills, cooperative learning strategies and assessments 
approaches lower, reflecting perhaps less project attention and focus on these 
areas, and/ or lower levels of initial knowledge of the areas. 
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Table 17 

MSA Teacher Self-Assessment 



2002 - 2003 
N=43 

Please rate yourself along the following dimensions as a 

result of your participation in MSA. Mean 



(SD) 



a) 


Knowledge/ understanding of your content area 


4.3 




(math, science, language arts, or social studies) 


(0.7) 


b) 


Knowledge/understanding of your content 


4.1 




standards (math, science, language arts, or social 
studies) 


(0.6) 


c) 


Confidence in teaching content area 


4.3 






(0.7) 


d) 


Knowledge of a wide variety of instructional 


4.2 




techniques 


(0.7) 


e) 


Technology skills 


3.8 






(1.0) 


f) 


Knowledge and implementation of cooperative 


3.9 




learning strategies (i.e., jigsaw, small groups) 


(•9) 


s) 


Knowledge of various assessment strategies 


3.7 






(0.8) 


h) 


Implementation of various assessment strategies 


3.6 






(0.8) 



Note. Scale: l=weak, 3= moderately strong, 5=very strong, NA=Not applicable. 



Teacher comments about the ways in which MSA impacted their 
knowledge are found below: 

Teacher 16 

I'm still in the learning & understanding process of standards and 
benchmarks. I have learned quite a lot of more about them and 
understand them a whole lot more as an MSA participant. 

Teacher 8 

As the 4 th grade teacher MSA has improved my knowledge of Math & 
Science standards and of resources to teach math and science. 
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Teacher 12 

I have a better understanding of my content areas and content standards. I 
now have resources to get to and information to get resources to help me 
teach my content areas and standards. 

Conclusion 

Year 4 of MSA was a year of expansion, with respect to the number of 
teachers, schools and districts involved with the project, and the ways in which 
the project was implemented. It was also a year of experimentation with 
cognitive coaching. Veteran MSA teachers took on the task of working with their 
colleagues in a "coaching setting" to continue the processes critical to on-going 
professional development. Previous years of MSA have demonstrated the 
importance of structured, facilitated interactions that focus specifically on 
teachers' instructional strategies as both a tool for accountability and as a means 
through which teachers can reflect on the nature and purpose of the approaches 
they use in classrooms. The refinements and revisions in the MSA model that 
occurred in previous years have strengthened MSA into a model for on-going 
professional development. The Math and Science Academy professional 
development model embodies strategies proven to improve teaching and 
learning (Darling-Hammond & McLaughlin, 1995). The MSA model continues to 
be: 



• Experiential in nature; teachers are given multiple opportunities to try out new 
ideas and reflect on those experiences. 

• Theoretical, research driven; MSA provides teachers with access to resources 
and ideas that are research-based, current, and well documented. 

• Collaborative: MSA teachers work in teams at their schools, across schools and 
as a project. 

• Oriented towards coaching, mentoring, and problem solving; the cognitive 
coaching model, with multiple opportunities for reflection continues as 
centerpiece of MSA. 

• Sustained, intensive and on going; each successive year, MSA has developed, 
grown and changed to meet the needs of participants by incorporating findings 
from what works and what doesn't in project classrooms. 
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Year 4 of the Math and Science Academy was a year of challenges and 
successes, of rethinking, revising and revamping ideas, understandings, and 
conceptualizations. A powerful model for professional development has 
emerged, one that encompasses the most important and essential elements of 
providing quality instruction for students of all ages. MSA continues to provide 
what research indicates is essential to quality reform to be sustained and to grow; 
the context, the expectation and the opportunities for teachers to learn what they 
need to know and practice those skills in a reflective, continually improving 
manner. 



Recommendations 

The recommendations that follow are based on project goals for Year 4, on 
teacher comments, observations and interviews with program participants. The 
recommendations are organized around project goals, with additional 
recommendations focused on project logistics. 



Planning 

Continue to guide teachers and teams at schools in the process of 
identifying learning goals, aligning those goals with standards, and revisiting 
those goals throughout the year. Teachers benefit from strategies to develop 
"living" documents that are easily accessible, easy to use, and meaningful to 
their instruction. As in previous years, it would be beneficial to include both 
content specific as well as cross-curricular planning activities. Teachers benefit 
from extended planning sessions, with specific, measurable outcomes. 

Collaboration 

Continue to offer teachers multiple opportunities to collaborate in formal 
and informal settings. "Show and tell," along the lines of "Celebrations of 
Learning," are powerful settings in which MSA teams, students, administrators, 
and parents can understand the way/ s in which project goals are being 
implemented and achieved. Collaboration at sites, through regularly scheduled 
after-school meetings, with teachers taking on increasing responsibility appears 
to be a promising model for using collaboration as a means to developing site 
leadership. 
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Instructional Strategies 

It is critically important to continue to help teachers understand, 
implement, and evaluate a variety of research-based instructional strategies, 
including technology. MSA provides teachers with important resources to new 
strategies. Careful reflection and selection of new ideas to incorporate is vital; too 
many goals and too many approaches can dilute MSA's impact long-term. 

The MSA 2004 Summer Institute is one example of supporting the 
development of teachers' knowledge of mathematics. Continuation and follow- 
up of the strategies and approaches to teaching standards-based mathematics is 
important. Without follow-up and additional support for the ideas presented at 
the Summer Institute, widespread use and implementation of the approaches 
may be limited. Teachers also requested additional content information in the 
area of science. 

Assessment 

MSA mentors and a number of MSA teachers recognize the importance of 
strengthening their knowledge and understandings of assessment to increase 
student learning. Absent specific instruction about assessment, it is challenging 
and almost impossible for teachers to build the kinds of assessment tools they 
need to more effectively support teaching and learning in their classrooms. A 
focus on quality assessment systems, and how to develop, implement, and 
understand student understandings is an important next step in the evolution of 
MSA. 

Classroom Management 

Important steps have been made towards strengthening teachers' capacity 
to successfully implement classroom management strategies. As teachers become 
more familiar with various approaches to classroom management, it is important 
to continue to communicate successes and challenges, and to refine classroom 
strategies to fit a variety of contexts. 

Cognitive Coaching Model 

During Year 4 of MSA, experienced MSA teachers took on roles and 
responsibilities in the coaching process at their sites. This step occurred without 
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formal instruction or direction; teachers, who had been coached previously, 
implemented the protocol in ways that best fit their particular site and situation. 
During the 2004 Summer Institute, coaches had the opportunity to learn about 
the coaching process in-depth, to ask questions about the nature of the reflection, 
and to refine the coaching protocol. To maximize the power of the coaching 
model, it is important to "calibrate" observations, that is, to generate more 
systematic guidelines for how and in what ways the 3-part coaching experience 
is implemented. Meetings and debriefings can be planned to support coaches in 
their quest to build their understandings of the coaching process. Mentors too, 
can benefit from careful articulation of project goals and ideas, as they work with 
the new coaches and teachers to implement MSA during Year 5 of the project. 

MSA Project Logistics 

MSA has grown and developed each successive year of the project and, 
with it, the roles and responsibilities of project leaders. In addition to providing 
critical support to teachers in the classroom, organizing and conducting Summer 
Institutes, and sessions during the school year, mentors have assumed numerous 
additional roles and responsibilities for running the project, in particular with 
respect to program logistics, such as stipend payment, contracts with districts 
and the like. Careful consideration of the best and most efficient ways to allocate 
mentor time and energy is a critical issue for Year 5 of the project. The addition 
of a 4th mentor will help to ensure that teachers will continue to receive the 
instructional support they need to implement project goals and objectives, and 
with planning and delivery of the Summer Institutes, but additional personnel 
and resources may be necessary for MSA to continue to grow and develop in quality 
ways. 



Implications 

During Year 4 of MSA, teachers and mentors continued to refine and 
develop project goals and objectives. As in previous years, project "success" was 
most dramatic in classrooms where teachers most fully implemented project 
goals and strategies. MSA teachers continue to make important strides towards 
refining their teaching practices and implementing the instructional strategies, 
methods, and tools to support student learning and achievement. 
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Appendix A 



Al. Teacher Survey 

A2. Student Achievement Data 
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1. Sex: 



2. Ethnicity 



[ ] White 

i ] African American 



[ ] Latino/a, Hispanic [ ] Native American 
[ ] Asian [ ] Other 



Academic/Professional Background 



3. What is the highest degree you have received? 

[ ] Bachelor's + Teaching credential 
[ ] Bachelor's + credential + units beyond 
[ ] Master's 



[ ] Master's + units beyond 
[ ] Doctorate 

[ ] Other (specify) 



4. Please indicate which teaching credentials you have and specify the content area of specialization. 
(Circle ALL that apply.) 

[ ] General Elementary [ ] Single Subjects 

[ ] General Secondary [ ] Bilingual 

[ ] Special Emergency [ ] Other 

[ ] Multiple Subject 



a. How many years of teaching experience do you have? 

b. How many years have you been a part of MSA? 1 year 

3 years _4 years 



_2 years 



6. How many years have you taught bilingual/LEP/bicultural students (including this year)? 

years 

7. Have you participated in other reform projects like MSA? [ ] Yes [ ] 

If yes, please describe: 



3. Please describe MSA classes you worked with this year: 

a. Grade/s: 

b. Subject (if applicable): science 

language arts 

Language(s) of instruction: 

1. Mostly Spanish 2. Boi 

3. Mostly English 4. Otl 



social studies 



2. Both English and Spanish 
4. Other 
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Planning an Effective Program: Curriculum and Articulation 



10. Please respond to the following statements based on your implementation of MSA ideas: 





Never 


Some- 

times 




Always 


N/A 


a) 


I develop yearlong and short-term 1 

goals for my students. 


2 


3 


4 


5 


N/A 


b) 


I choose content to meet the 
learning goals of my students. 






A 




N/A 




1 


3 


4 


5 


c) 


I design and adapt curricula to , 

meet the needs of my students. 


2 


3 


4 


5 


N/A 


d) 


I use instructional strategies that 
develop and promote student 
understanding. 


2 


3 


4 


5 


N/A 


e) 


I work with my colleagues in my 
content area/grade level to set goals 1 

and standards for learning and 
achievement. 


2 


3 


4 


5 


N/A 


f) 


I work with my colleagues across 
content areas to set goals and 


2 


3 


4 


5 


N/A 




standards for learning and 
achievement. 












g) 


I developed goals for “data not 
guesswork” performance and used 
them to guide instruction. 


2 


3 


4 


5 


N/A 



11. To what extent has your instructional planning, articulation and collaboration with your colleagues 
changed as a result of your participation in MSA? 



Not at 

All 

1 2 



A Great 

Somewhat Deal 

3 4 5 



Please explain. 
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Guiding and Facilitating Learning 



12. Please respond to the following statements based on your implementation of MSA ideas: 







Never 




Some- 

times 




Always 


N/A 


a) 


I focus and support inquiry as I 
interact with my students. 


1 


2 


3 


4 


5 


N/A 


b) 


I support student discussion of 
ideas in small and large groups. 


1 


2 


3 


4 


5 


N/A 


c) 


I model and provide guidelines for 
positive ways to share ideas and 
information. 


1 


2 


3 


4 


5 


N/A 


d) 


I require students to take 
responsibility for their learning and 
to work collaboratively. 


1 


2 


3 


4 


5 


N/A 


e) 


I recognize and respond to student 
diversity. 


1 


2 


3 


4 


5 


N/A 


f) 


I expect all students to participate 
fully in learning. 


1 


2 


3 


4 


5 


N/A 



13. To what extent have your approaches to guiding and facilitating student learning changed as a result of 
your participation in MSA? 



Not at 

All 

1 2 



A Great 

Somewhat Deal 

3 4 5 



Please explain. 
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Assessing Student Learning 



14. Please respond to the following statements based on your implementation of MSA ideas: 







Never 




Some- 

times 




Always 


N/A 


a) 


I systematically gather data on my 
students and their learning in my 
classes. 


1 


2 


3 


4 


5 


N/A 


b) 


I analyze assessment data on a 
regular and timely basis to inform 
and guide my teaching. 


1 


2 


3 


4 


5 


N/A 


c) 


I guide my students in self- 

assessment. 


1 


2 


3 


4 


5 


N/A 


d) 


I use student data, observations of 
teaching, and interactions with 
colleagues to reflect on and improve 
my teaching practices. 


1 


2 


3 


4 


5 


N/A 


e) 


I provide students with information 
on how their work will be assessed. 


1 


2 


3 


4 


5 


N/A 


f) 


I provide students with examples and 
models of what represents “good 
work”. 


1 


2 


3 


4 


5 


N/A 


g) 


I use a wide variety of assessments 
to help me understand students’ 
ideas and learning. 


1 


2 


3 


4 


5 


N/A 



15. To what extent have your assessment practices for teaching and learning changed as a result of your 
participation in MSA? 



Not at 

All 

1 2 



A Great 

Somewhat Deal 

3 4 5 



Please explain. 
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Instructional Materials and Resources 



16. Please respond to the following statements based on your implementation of MSA ideas: 







Never 




Sometimes 




A Great 
Deal 


N/A 


a) 


I structure learning time to allow 
students to engage in projects 
and/or investigations. 


1 


2 


3 


4 


5 


N/A 


b) 


I create a setting for student work 
that is flexible and supportive of 
student learning. 


1 


2 


3 


4 


5 


N/A 


c) 


I ensure a safe learning 
environment. 


1 


2 


3 


4 


5 


N/A 


d) 


I make tools & materials available 
to students to support learning. 


1 


2 


3 


4 


5 


N/A 


e) 


I make print resources available to 
students to support learning. 


1 


2 


3 


4 


5 


N/A 


f) 


I make technological resources 

available to students to support 
learning. 


1 


2 


3 


4 


5 


N/A 


h) 


I use graphic organizers to support 
learning. 


1 


2 


3 


4 


5 


N/A 


g) 


I engage students in designing their 
learning assignments. 


1 


2 


3 


4 


5 


N/A 



17. To what extent has your design and management of students’ learning environment changed as a result 
of your participation in MSA? 



Not at 

All 

1 2 



A Great 

Somewhat Deal 

3 4 5 



Please explain. 
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Building Communities of Learners 



18. Please respond to the following statements based on your implementation of MSA ideas: 







Never 




Some- 

times 




Always 


N/A 


a) 


I display and encourage respect for 
the ideas, skills and experiences of 
my students. 


1 


2 


3 


4 


5 


N/A 


b) 


I give students a voice in decisions 
about the content and context of their 
work. 


1 


2 


3 


4 


5 


N/A 


c) 


I require students to take 
responsibility for the learning of all 
members of their group/class. 


1 


2 


3 


4 


5 


N/A 


d) 


I support collaboration among my 
students. 


1 


2 


3 


4 


5 


N/A 


e) 


I structure and facilitate ongoing 
formal and informal discussions 
based on a shared understanding of 
the rules of classroom discourse. 


1 


2 


3 


4 


5 


N/A 


f) 


I model and emphasize the skills, 
attitudes, and values of inquiry. 


1 


2 


3 


4 


5 


N/A 



19. To what extent have your ideas and practices relating to the development of learning communities with 
your students changed as a result of your participation in MSA? 



Not at 

All 

1 2 



A Great 

Somewhat Deal 

3 4 5 



Please explain. 
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School & MSA Community 



20. Please respond to the following statements based on your implementation of MSA ideas: 







Never 




Some- 

times 




Always 


N/A 


a) 


I participate in planning and 
developing the school program for 
my content area/grade level. 


1 


2 


3 


4 


5 


N/A 


b) 


I have a voice in making decisions 
regarding the allocation of time and 
other resources at my school. 


1 


2 


3 


4 


5 


N/A 


c) 


I plan and implement professional 
growth and development strategies 
for myself and my colleagues. 


1 


2 


3 


4 


5 


N/A 


d) 


I communicate with the parents in 
our community about MSA goals 
and student progress towards those 
goals. 


1 


2 


3 


4 


5 


N/A 


e) 


I meet with my MSA colleagues to 
discuss student work, teaching and 
learning on a regular basis. 


1 


2 


3 


4 


5 


N/A 



21. To what extent have you changed your involvement and participation in the ongoing planning and 
development of the school learning plan as a result of your participation in MSA? 



Not at 








A Great 


All 




Somewhat 




Deal 


1 


2 


3 


4 


5 



Please explain: 
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MSA Program Effectiveness 



22. How effective was MSA in the following areas: 







Not 

Effective 




Some- 

what 

Effective 




Highly 

Effective 


N/A 


a) 


Familiarizing you with standards- 
based instruction 


1 


2 


3 


4 


5 


N/A 


b) 


Developing your knowledge of state 
frameworks for content areas 


1 


2 


3 


4 


5 


N/A 


c) 


Helping you develop 
interdisciplinary curriculum units 


1 


2 


3 


4 


5 


N/A 


d) 


Providing demonstration lessons that 
were meaningful and relevant 


1 


2 


3 


4 


5 


N/A 


e) 


Sharing assessment strategies 


1 


2 


3 


4 


5 


N/A 


f) 


Helping you to develop rubrics to 
support instruction 


1 


2 


3 


4 


5 


N/A 


g) 


Informing/involving the community 
about MSA goals and objectives 


1 


2 


3 


4 


5 


N/A 


h) 


Helping you understand how to use 
technology effectively 


1 


2 


3 


4 


5 


N/A 


i) 


Assisting you in implementing 
cooperative learning activities 


1 


2 


3 


4 


5 


N/A 



Comments: 
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MSA Impact: Self-Assessment 



23. Please rate yourself along the following dimensions as a result of your participation in MSA. If you 
teach more than one content area, please use the comment area below to indicate your self-assessments of 

Question 23a and Question 23b. 







Weak 




Moderately 

strong 




Very 

strong 


N/A 


a) 


Knowledge/understanding of your 
content area (math, science, 
language arts, or social studies) 


1 


2 


3 


4 


5 


N/A 


b) 


Knowledge/understanding of your 
content standards (math, science, 
language arts, or social studies) 


1 


2 


3 


4 


5 


N/A 


c) 


Confidence in teaching content area 


1 


2 


3 


4 


5 


N/A 


d) 


Knowledge of a wide variety of 
instructional techniques 


1 


2 


3 


4 


5 


N/A 


e) 


Technology skills 


1 


2 


3 


4 


5 


N/A 


f) 


Knowledge and implementation of 
cooperative learning strategies (i.e., 
jigsaw, small groups) 


1 


2 


3 


4 


5 


N/A 


g) 


Knowledge of various assessment 
strategies 


1 


2 


3 


4 


5 


N/A 


h) 


Implementation of various 
assessment strategies 


1 


2 


3 


4 


5 


N/A 



24. Comments: 



Student Learning and Achievement 



25. Please indicate your observations regarding student learning and achievement this year. 







Disagree 




Moderately 

strong 




Strongly 

Agree 


N/A 


a) 


I have observed changes in student 
learning and achievement this year. 


1 


2 


3 


4 


5 


N/A 


b) 


My participation in MSA had a 
positive impact on my students’ 
learning and achievement this year. 


1 


2 


3 


4 


5 


N/A 


c) 


MSA ideas helped increase student 
learning and achievement. 


1 


2 


3 


4 


5 


N/A 


d) 


Students learned more because of 
cooperative learning opportunities in 
my classroom. 


1 


2 


3 


4 


5 


N/A 


e) 


Students benefited from the use of 
DNG in my classroom. 


1 


2 


3 


4 


5 


N/A 
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26. Please explain and provide specific examples. 



27. Coaching Experience 

Describe how the pre-conference, observation and debriefing experience with coach/es impacted (or 
not) the teaching, learning and planning in your classroom this year. Please include specific 
examples if possible. 



28. Data Not Guesswork 

Describe how and in what ways you used the “data not guesswork” to guide instruction this year (if 
at all). Please include specific information on the structure you used, types of questions developed, 
frequency of use and overall reactions to this instructional/assessment approach. 



29. Did you respond to the weekly informational e-mail messages? 



Not at All 



Sometimes 



Almost Always 



1 



2 



3 



4 



5 



How useful was the information sent to you as a professional? 



Not at all useful 




Somewhat useful 




Highly useful 


1 


2 


3 


4 


5 



Please provide more information on your response to Question #27: 



30. For Site Leaders: describe your experience working with your team members this year. How (if at 
all) did your leadership role impact your experience with MSA, your teaching and your relationship 
with your colleagues? 
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31. List three successes in the implementation of MSA at your school site. Please provide details and 
examples. 

1) 

2) 

3) 



32. List three barriers to the implementation of MSA at your school site. Please provide details and 
examples 

1 ) 

2) 

3) 

33. How could MSA be improved? 

Thank you for completing this survey. 
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Appendix A2 

Student Achievement Scores 



Table 7 

School A: 7th Grade 
MSA CTB/Scale Scores 





Reading/SS 


Language 

Arts/SS 


Math/SS 


Science/SS 


Social 

Studies/SS 


Year 












2000 


634 


634 


612 


628 


636 


2001 


641 


642 


621 


640 


642 


2002 


648 


649 


636 


645 


642 


2003 


674 


672 


665 


673 


666 


2004 


662 


667 


658 


671 


677 


Score 
change 
‘03 to ‘04 


-12 


-5 


-7 


-2 


+11 



Table X 

School B: 7 th Grade 
MSA CTB/Scale Scores 





Reading/ SS 


Language 

Arts/SS 


Math/SS 


Science/SS 


Social 

Studies/SS 


Year 












2000 


631 


634 


623 


629 


634 


2001 


642 


642 


634 


633 


638 


2002 


646 


646 


642 


644 


646 


2003 


658 


659 


656 


664 


656 


2004 


659 


663 


661 


663 


662 


Score 

change 

from 

03-04 


+1 


+4 


+5 


-1 


+6 
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